Description

PR to apply for E2E OLS evaluation framework for AAP chatbot

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Create a virtual environment, and install the necessary package via make install-deps, for this user need to clone ansible-chatbot-service repo.
• For running the E2E tests locally, user need to configure olsconfig.yaml and copy to parent directory, for AAP chatbot scenario sample olsconfig.yaml is under /scripts/evaluation/ folder.
• Once configured, user can run the evaluation framework over AAP complete set of QnA as under aap_doc_qna.parquet under /scripts/evaluation/eval_data/ folder, or else run over sample QnA defined under scripts/evaluation/eval_data/aap-sample.parquet file.
• To run the eval framework, run following cmd:

OPENAI_API_KEY=IGNORED python -m scripts.evaluation.driver --qna_pool_file /Users/sjaiswal/Sumit/wisdom/ansible-chatbot-service/scripts/evaluation/eval_data/aap-sample.parquet --eval_provider_model_id my_rhoai+granite3-8b --eval_metrics answer_relevancy answer_similarity_llm cos_score rougeL_precision --eval_modes ols_rag --judge_model granite3-8b --judge_provider my_rhoai --eval_query_ids qna1 qna2 qna3 qna4 qna5